ANNENBERG 


BROWN UNIVERSITY 


=to NYAVCol a dlale) at-) ol-1am \ oh-4 Oey-7-4 


Projecting the potential impacts of COVID-19 
school closures on academic achievement 


Megan Kuhfeld James Soland Beth Tarasawa 
NWEA University of Virginia NWEA 

Angela Johnson Erik Ruzek Jing Liu 

NWEA University of Virginia Brown University 


With 55 million students in the United States out of school due to the COVID-19 pandemic, 
education systems are scrambling to meet the needs of schools and families, including planning 
how best to approach instruction in the fall given students may be farther behind than in a typical 
year. Yet, education leaders have little data on how much learning has been impacted by school 
closures. While the COVID- 19 learning interruptions are unprecedented in modern times, existing 
research on the impacts of missing school (due to absenteeism, regular summer breaks, and school 
closures) on learning can nonetheless inform projections of potential learning loss due to the 
pandemic. In this study, we produce a series of projections of COVID-19-related learning loss and 
its potential effect on test scores in the 2020-21 school year based on (a) estimates from prior 
literature and (b) analyses of typical summer learning patterns of five million students. Under these 
projections, students are likely to return in fall 2020 with approximately 63-68% of the learning 
gains in reading relative to a typical school year and with 37-50% of the learning gains in math. 
However, we estimate that losing ground during the COVID-19 school closures would not be 
universal, with the top third of students potentially making gains in reading. Thus, in preparing for 
fall 2020, educators will likely need to consider ways to support students who are academically 
behind and further differentiate instruction. 


VERSION: May 2020 


Suggested citation: Kuhfeld, Megan, James Soland, Beth Tarasawa, Angela Johnson, Erik Ruzek, and Jing Liu. 
(2020). Projecting the potential impacts of COVID-19 school closures on academic achievement. 
(EdWorkingPaper: 20-226). Retrieved from Annenberg Institute at Brown University: 
https://doi.org/10.26300/cdrv-yw05 


Projecting the potential impacts of COVID-19 school closures on academic achievement 


Megan Kuhfeld 
Research Scientist, NWEA 


James Soland 
Assistant Professor, Curry School of Education University of Virginia 


Beth Tarasawa 
Executive Vice President of Research, NWEA 


Angela Johnson 
Research Scientist, NWEA 


Erik Ruzek 
Research Assistant Professor, Curry School of Education University of Virginia 


Jing Liu 
Postdoctoral Research Associate, Annenberg Institute, Brown University 


Projecting the potential impact of COVID-19 school closures on academic achievement 


Abstract 


With 55 million students in the United States out of school due to the COVID-19 pandemic, 
education systems are scrambling to meet the needs of schools and families, including planning 
how best to approach instruction in the fall given students may be farther behind than in a typical 
year. Yet, education leaders have little data on how much learning has been impacted by school 
closures. While the COVID-19 learning interruptions are unprecedented in modern times, 
existing research on the impacts of missing school (due to absenteeism, regular summer breaks, 
and school closures) on learning can nonetheless inform projections of potential learning loss 
due to the pandemic. In this study, we produce a series of projections of COVID-19-related 
learning loss and its potential effect on test scores in the 2020-21 school year based on (a) 
estimates from prior literature and (b) analyses of typical summer learning patterns of five 
million students. Under these projections, students are likely to return in fall 2020 with 
approximately 63-68% of the learning gains in reading relative to a typical school year and with 
37-50% of the learning gains in math. However, we estimate that losing ground during the 
COVID-19 school closures would not be universal, with the top third of students potentially 
making gains in reading. Thus, in preparing for fall 2020, educators will likely need to consider 


ways to support students who are academically behind and further differentiate instruction. 


Introduction 


Virtually all K-12 students in the United States had face-to-face instruction interrupted 
during the 2019-20 school year due to the SARS-CoV-2 (COVID-19) pandemic. The majority of 
school districts are providing some virtual instruction during the last months of the school year 
(Lake & Dusseault, 2020a). But it remains unclear how effective virtual learning will be, given 
that most K-12 students and teachers have little experience with online instruction and that large 
gaps in technology access exist in many parts of the country. Additionally, during the extended 
school closure, many working parents struggle to educate and care for their children. These 
unique educational challenges are accompanied by broader shocks to society, including a major 
economic downturn, job losses, and the tangible health threat that is COVID-19. In short, 
extended time out of school will almost certainly affect student achievement (likely in a negative 
way for many), and that impact is hard to estimate given all the unique aspects of COVID-19 on 


schooling and society. 


While many aspects of the pandemic make anticipating its impact on achievement 
difficult, there are parallels between the current situation and other planned and unplanned 
reasons for which students miss school that can help us quantify the potential scale of the 
COVID-19 impact. Specifically, existing research on the effects on learning of (a) summer 
vacation, (b) weather-related school closures (e.g., Hurricane Katrina in New Orleans), and (c) 
out-of-school time due to absenteeism can provide a rough sense of how additional time out of 
school due to COVID-19 will affect achievement in the coming fall and longer term. The intent 
of our study is to better understand and project how COVID-19-based school closures might 
affect achievement and growth during the current school year (2019-20) and the next (2020-21). 


Given that our projections, while based on existing literature, are unable to account for the 


impact of virtual instruction, access to supplemental curriculum, or the availability of additional 
educational resources, among other important factors, we present these results as preliminary 


estimates of the potential negative impacts expected due to extended school closures. 


Prior research on time students spend out of school is useful given the importance of 
forecasting the impact of COVID-19 on short- and long-term achievement. Teachers and schools 
can benefit from knowing not only how much lower achievement might be but also how much 
more variable it could be in the fall. If students begin school in the fall of 2020 (or whenever 
regular schooling resumes) with bigger gaps in content knowledge between low- and high- 
performing students, then strategies like expanding instructional differentiation may be 
warranted. Further, projections of how potential learning loss due to out-of-school time might 
affect growth in the coming school year may also help educators identify students who are not on 


track academically when school resumes and give them needed supports. 


In this study,! we made projections about the effects of COVID-19 on student 
achievement trends from the spring of 2020, when schools were first shut down across the 
United States (U.S.), through to the start of the 2020-21 school year. To provide preliminary 
estimates of the potential impacts of the extended pause on face-to-face academic instruction 
during the pandemic, we used a national sample of five million students in Grades 3-8 who took 
MAP® Growth™ assessments in the 2017-18 and 2018-19 school years (e.g., about 22% of the 
approximately 22 million U.S. public school students in Grades 3-8 according to NCES [2018]). 


Specifically, we compared typical growth trajectories across a standard-length school year to 


' This paper has its origins in a NWEA brief (Kuhfeld & Tarasawa, 2020), which presents some preliminary 
learning projections. The current paper is distinct from the brief in terms of the volume of analyses and theoretical 
grounding. 


learning projections that assume students are out of school for the last three months of the 2019- 


20 school year. In so doing, we investigated three research questions: 


(1) What are possible scenarios (based on prior literature and recent MAP Growth data) 
for student learning patterns during the 2019-20 school year as a result of the school 


closures? 


(2) How much variability do we expect in (a) students’ learning rates during the extended 
school closure period and (b) students’ fall 2020 scores assuming a normal 2019-20 


school year versus one disrupted by COVID-19? 


(3) What is the association between out-of-school time due to COVID-19 and projected 


subsequent learning rates over the course of the 2020-21 school year? 
Background 


While the COVID-19 school closures are unprecedented in the U.S., there are multiple 
bodies of research on which we can draw to anticipate the impacts” of extended closures on 
student learning. These include (a) seasonal learning studies that compare learning that occurs 
during the school year to learning that occurs during summer breaks, (b) studies on weather- 
related school closures, and (c) studies on student absenteeism. Table 1 provides a summary of 
the effect sizes (reported in standard deviation [SD] units for each day out of school) from key 
studies in each body of literature that are discussed below (further details on the studies are 


provided in Appendix A of the supplemental materials). We then discuss the degrees to which 


2 Studies from these three lines of research provide descriptive as well as credibly causal 
evidence. For the purpose of this study, we consider the research evidence collectively without 
distinguishing causal estimates from associations and refer to all estimated relations between out- 
of-school time and achievement as effects or impacts. 
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each of these bodies of work is likely to reflect the conditions observed during the COVID-19 


school closures. 
Seasonal Learning Studies 


Seasonal learning research (including studies to understand the effects of summer 
learning loss) makes comparisons of student learning patterns when school is in versus out of 
session. Thus, one way to think about COVID-19 school closures is to consider them extensions 
of summer break for most students. Research has consistently shown that achievement typically 
slows or declines over the summer months (on average) and that the declines tend to be steeper 
for math than for reading (Quinn & Polikoff, 2017). However, there is much debate about the 
magnitude of summer loss and the degree to which summer vacation contributes to 


socioeconomic achievement gaps (von Hippel, 2019). 


Prominent early work on summer learning loss found that students lost about a month of 
learning over the summer, with lower-income students falling behind middle- and high-income 
students in reading (Cooper, Nye, Charlton, Lindsay, & Greathouse, 1996; Alexander, Entwisle, 
& Olson 2001). Recent summer loss research using the Early Childhood Longitudinal Study, 
Kindergarten Cohort (ECLS-K) has indicated minimal loss on average during the summer, while 
studies using NWEA’s MAP Growth assessment showed fairly sizable drops (Atteberry & 
McEachin; 2020; Kuhfeld, Condron, & Downey, 2019). This variability in estimates can be seen 
in Table 1, where summer drop estimates range from 0.001 to 0.010 SDs per day of school 
missed across grades/subjects. However, research using both recent data sources agree that 
summer does not appear to be a time in which socioeconomic and racial/ethnic inequalities 


widen (e.g., von Hippel & Hamrock, 2019; Kuhfeld, 2019; von Hippel, 2019). 


School Closures due to Inclement Weather and Natural Disasters 


The literature on school closures also provides some insight into the potential effect of 
COVID-19 school closures, especially given such closures occur unexpectedly and disrupt 
scheduled instruction. Although they occur over a shorter duration, school closures resulting 
from inclement weather or natural disasters provide an analog to school closures due to COVID- 
19. Absent the weather event or natural disaster, schools would be in session and learning for 
most students would occur as normal. Hansen (2011) found that each day of school cancellation 
due to snow in Colorado reduced 8" grade math achievement by magnitudes ranging from 0.013 
to 0.039 SDs, and the impact effects of snow days in Maryland ranged from 0.013 to 0.016 SDs. 
Goodman (2014) studied snow day closures in Massachusetts and found that each day of school 
closure had null effects on math and reading achievement overall, but that students attending 
poor schools experienced a decline of 0.014 SDs in math and 0.016 SDs in reading for every day 
of school closure. A related line of research found that the displacement effect of Hurricane 
Katrina led to drops in achievement at a magnitude of approximately 0.10 SDs in the year after, 
though these studies did not investigate effect heterogeneity by student demographics or school 
poverty (e.g., Sacerdote, 2012). However, these estimates are not comparable to those provided 


by the snow day literature due to differences in research design and recorded units of time. 
Absenteeism 


In contrast to the seasonal learning and school closure studies discussed above, an 
emerging literature on school absenteeism focuses on the impact of instructional time loss due to 
absences while schools are in session. Unlike the school closure due to the COVID-19 that forces 
every student to be out of school, not all students are absent during a normal school year. There 
are numerous reasons for which a student might miss school, including lack of access to reliable 


transportation and need to care for family members. Minority and low-income students tend to 


have more absences and are more likely to be chronically absent (i.e., missing at least 10% of 


school days), compared with their more affluent peers (Whitney & Liu, 2016). 


Research consistently found that absences had negative effects on end-of-year test scores. 
Several studies that used a value-added model found similar effect sizes in both elementary and 
secondary schools. Specifically, missing ten school days can decrease student math test scores by 
0.06 to 0.08 SDs; the effect sizes for ELA scores were slightly smaller (Aucejo & Romano, 
2016; Gershenson, Jacknowitz, & Brannegan, 2017; Liu, Lee, & Gershenson, 2019). Studies that 
used either flu or snow days as an instrumental variable for absences tended to yield much larger 
estimates (Aucejo & Romano, 2016; Goodman, 2014) largely due to the specific variation used 
in estimating the impact of absences. For example, Goodman (2014) found that one moderate 
snow day-induced absence reduced student math scores by 0.05 SDs. Another takeaway from the 
absenteeism literature is that the negative effects of absences were linear, meaning that each 
additional absence caused similar learning loss no matter how many absences a student had 


already accrued (Gershenson et al., 2017; Liu et al., 2019). 


Similarities/Differences Between Out-of-School Time Studies and COVID-19 School 


Closures 


The literatures on summer vacation, school closures due to weather and natural disasters, 
and absenteeism indicate that student learning is likely to be negatively impacted by being out of 
school. While there is a fair amount of variability in the effect size estimates by grade and study 
(Table 1), some clear trends emerge. Students showed bigger losses in math than reading while 
out of school. Being absent from school is generally associated with larger impacts on learning 
than being out of school due to summer vacation, particularly in middle school. Finally, our 


review suggests that studies on summer loss and absenteeism may provide better (if imperfect) 


models for the impact of COVID-19 than the literature on weather-related school closures, which 
was sparse (only two studies with effect size estimates), generated inconsistent findings, and 
tended to rely on small sample sizes from specific geographical settings. Accordingly, we draw 
on the absenteeism effect sizes reported in Table 1, as well as new summer loss analyses, to 


produce the projections reported in this study. 


Before describing our approach, we consider how current and past school closures and 
their impact on achievement may differ. First, relying on past precedent may overstate the effect 
of COVID-19 school closures. Specifically, the biggest difference between school closures 
examined by previous studies and those of COVID-19 is that most school districts are now 
providing online instruction. Many districts have offered remote learning plans, which may 
include formal curriculum, assignments, and/or progress-monitoring as well as access to general 
educational resources. By April 3™ - 4", 83 percent of parents in a Gallup poll indicated their 
child was involved in an online learning program from their school (Brenan, 2020). Further, one 
could imagine that parents of high socioeconomic status (SES) might leverage their cultural 
capital such that their children actually make larger academic gains than in typical school days, 


and these gains could further contribute to educational disparities. 


Second, there is also evidence suggesting that measures taken by schools may not be as 
effective as hoped. There are concerning signs that many teachers have had no contact at all with 
a significant portion of students (Lieberman, 2020). According to national survey of teachers 
conducted by EdWeek (Kurtz, 2020), as of April 8" only 39% of teachers reported interacting 
with their students at least once a day, and most teacher-student communication occurred over 
email. There is also evidence that, even when teachers are making themselves and their 


instructional materials available virtually, many students lack the means to access online 


materials from home. Nearly 50% of low-income families and 42% of families of color lack 
sufficient devices at home to access distance learning, according to an Education Trust (2020b) 
poll. Moreover, few school systems provide plans to support students who need accommodations 
or other special populations (Lake & Dusseault, 2020b). Thus, despite many administrative 
leaders’ and educators’ best efforts, students and their families may bear the brunt of the 


responsibility for ensuring learning continues during the closures. 


There is also uncertainty about whether virtual instruction, even when well-implemented, 
is likely to be as effective as traditional face-to-face instruction. Prior comparisons of online and 
traditional public schools show that students in online schools lose between 0.1 and 0.4 SDs on 
standardized tests compared to students in traditional schools (Gill et al. 2015; CREDO, 2015; 
Ahn & McEachin, 2017). The COVID-19 virtual instruction is somewhat different because 
students already know their teachers and are potentially doing review rather than being taught 
new material. However, many public teachers have not been trained on how to provide effective 


virtual instruction. 


Finally, past precedent on out-of-school time may understate the impact of COVID-19 on 
student learning, especially compared to summer break, which is a wholly anticipated event. The 
same Education Trust (2020b) poll of California and New York parents found that elevated stress 
levels for families (parents and children) continue due to economic uncertainty and job loss, 
fears about catching a life-threatening virus, and the psychological impact of social isolation and 
disruptions to everyday life. The (almost certainly adverse) effect of these economic and 
psychological factors on the learning occurring in homes is difficult to anticipate. However, 
extended school closures due to natural disasters such as Hurricane Katrina and the Christchurch, 


New Zealand earthquakes may provide some clues. Research suggests the impact of school 
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disruptions following natural disasters on student development was long lasting, with some 
students continuing to show psychological distress and trouble concentrating for several years 


afterwards (Picou & Marshall, 2007; Duncan, 2016). 


Given unique elements of the current situation, we are not positioned in this study to 
speculate about whether current research and historical trends in achievement will likely 
understate or overstate the effects of COVID-19 school closures on achievement. However, 
given the scale of our data and what we know from past research, we can make forecasts about 
potential impacts of COVID-19 based on multiple scenarios and assumptions about how learning 
might have changed this past school year (2019-20) and will change over the next (2020-21). 
Even if forecasts can only provide a range of potential impacts based on different assumptions 
made about the current situation, forecasts are nonetheless invaluable in helping educators and 
policymakers understand what to expect when students return in the fall, including how learning 


might progress differently over the course of the 2020-21 school year. 


To that end, our study includes several analyses that can prepare educators and 
policymakers for what they may face next year. First, we produce two sets of possible scenarios 
for COVID-19 learning loss while students would have otherwise been in school in 2019-20. 
One set of projections is based on empirical analyses examining summer loss using MAP 
Growth data. We then compare those projections to a second set of projections for learning loss 
based on the absenteeism literature, obtained by multiplying the daily learning loss rate from that 
literature by the days of school missed during the pandemic. Second, we provide estimates of (a) 
predicted variability in learning rates and (b) predicted variability in student scores at the 
beginning of the 2020-21 school year that account for the extended time out of school. Third, we 


go beyond prior school closure research to look not only at the potential effect of school closure 
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on current achievement, but also the relationship between out-of-school time achievement 
declines and growth during the following year (i.e., how strongly associated is the magnitude of 


learning loss with the gains made in the next year?). 
Methods 
Analytic Sample 


The data for this study are from NWEA’s anonymized longitudinal student achievement 
database. School districts use NWEA’s MAP Growth assessments to monitor elementary and 
secondary students’ reading and math growth throughout the school year, with assessments 
typically administered in the fall, winter, and spring. We use the test scores of approximately five 
million third- to seventh-grade students? in 18,958 schools across the United States. In this study, 
we follow students across two school years (2017-18 and 2018-19) and one summer break 
(summer of 2018). The NWEA data also include demographic information, including student 
race/ethnicity, gender, and age at assessment, though student-level SES is not available. Table 2 
provides descriptive statistics for the sample by subject and grade. Overall, the sample is 51% 
male, 47% White, 17% Black, 4% Asian, and 18% Hispanic. School-level free or reduced priced 
lunch (FRPL) eligibility was obtained from the 2017-18 Common Core of Data (CCD) file from 
the National Center of Education Statistics (NCES). The average student in our sample attends a 
school that is 51% FRPL-eligible. A comparison of the 18,972 schools in our sample relative to 
U.S. population of public elementary and middle schools (72,075 schools serving Grades 3-8) is 


provided in Appendix B of the supplemental materials. Overall, the sample closely aligns to the 


3 Due to limited MAP Growth testing in high schools, we did not follow the cohort of 8" graders in 2017-18 into 9% 
grade in 2018-19. 
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characteristics of U.S. public schools, with a slight overrepresentation of Black students and 


underrepresentation of Hispanic students. 
Measures of Achievement 


Student test scores from NWEA’s MAP Growth reading and math assessments are used 
in this study. MAP Growth is a computer adaptive test that precisely measures achievement even 
for students above or below grade level and is vertically scaled to allow for the estimation of 
gains across time. The MAP Growth assessments are typically administered three times a year 
(fall, winter, and spring) and are aligned to state content standards. Test scores are reported on 
the RIT (Rasch unIT) scale, which is a linear transformation of the logit scale units from the 


Rasch item response theory model. 
Projecting COVID-19 School Closure Impacts on Learning Trajectories 


In this study, we present two sets of estimates of the potential impacts of COVID-19 
school closures on student learning: (a) empirical estimates calculated using MAP Growth data 
based on summer loss patterns during the summer of 2018, and (b) estimates calculated based on 
prior absenteeism literature. We begin by describing our empirical approach to estimating 
students’ academic growth during the school year and learning loss during summer break under 
normal (pre-COVID-19) conditions. Subsequently, we discuss how we use the absenteeism and 


summer loss estimates to produce COVID-19 projections. 


We first estimated typical growth rates across two school years (2017-18 and 2018-19) 
and the summer break in between using a series of multilevel growth models (longitudinal test 
scores nested within students within schools). Following other seasonal learning research studies 
(e.g., von Hippel et al., 2018; Kuhfeld et al., 2019), we estimated student learning rates as a 
function of the months that elapsed during the two school years and the summer between. Given 
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that prior research using MAP Growth data found evidence of non-linearity in students’ within- 
school growth trajectories (Kuhfeld & Soland, 2020), particularly in reading, we modeled 
student learning rates across the school year using a quadratic function (though a set of models 
assuming linear growth are also reported in Appendix Tables C3 and C4). Under this model, the 
test score y;;; for student 7 in school j at timepoint ¢ was modeled as a quadratic function of the 
months that a student had been exposed to the 2017-18 school year (MonY1;; ), the summer of 
2018 (Sum;;), and the 2018-19 school year (MonY2;;). At level 1, the growth model can be 
expressed as: 


Vij = TN oij + 74;; MonY 1, ;; + 1p;;MonY1;;; + 134; SUM}; ; (1) 


+1 4;; MonY2,;; + Ts; MonY2¢;; + Ctij: 


The intercept (7;;) 1s the predicted score for student i in school j tested on the first day of the 
2017-18 school year, 774;; is the average instantaneous rate of change at the start of the 2017-18 
school year, and 73;; is the average rate of change of the linear growth term in 2017-18 for a 
one-month change in time (e.g., the acceleration or deceleration in growth), 773;; is the monthly 
summer linear loss rate, and 74;; and 75;; are the linear and quadratic terms in the 2018-19 
school year, respectively. At level 2 and 3 of the model, the intercept and growth parameters 


were allowed to vary among students within schools and between schools: 


Level-2 Model (student (i) within school (j)): (2) 
Toij = Booj + Toi; 
mij = Proj + Nij 
12ij = B20; 
13ij = B30; + Psi; 
m4ij = Baoj + Taij 
Tsij = Bsoj 
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Level-3 Model (school (j)): 
Booj = Yooo + Ugo; 
Boj = Y100 + U0; 
B20; = Y200 
B30; = Y300 + U3z0; 
Boj = Yaoo + Ugo; 


Bsoj = Ys500 


Variance component specification: 


Ctis ~ N(0, Ohis)> ris~MVN(O, Ts), u,~MVN(0, T sch). 


This model was estimated separately by subject (math and reading) and grade (3-7) using HLM 
Version 7 (Raudenbush, Bryk, & Congdon, 2013). Estimated parameters from these models are 


reported in Appendix Tables C1 and C2. 


We began by calculating “typical” growth rates across a standard 9.5-month school year 
(assuming students start school on September 1‘ and end on June 15"). To estimate typical 
growth, we used the estimated parameter estimates from the 2017-18 school year for each grade 


g and subject separately: 
RIT rg = Poor + (P100) * Mon; + (7200) * Moné, (3) 


where Mon; takes values from 0 to 9.5. We then calculated “typical” summer loss across a 2.5- 


month summer: 
SumLoss;, = (7300) * SumMon,, (4) 


where SumMon, takes values from 0 to 2.5 months. Under the standard-length school year, 


students end the year at their 9.5-month achievement level (RIT s g) and then were assumed to 
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lose ground linearly across a 2.5-month summer. We provided the “typical” school year growth 


rates and summer loss as a reference for the COVID-19 projections described below. 


The first scenario, which we refer to as “COVID Loss Summer Slide”, assumes that 
assumes that typical summer loss patterns would extend through the prolonged school closure. 
Linear projections were made based on the same SumLos$syg calculation described above, but 
starting from the projected achievement level at 6.5 months (RIT, 5 g) and extending to the 
presumed start of the next school year (12 months, September 1st). During the “normal” summer 
period (9.5 to 12 months), the typical summer loss and COVID Loss Summer Slide rates were 


the same, and so these lines were parallel during the summer months (June 15" to September 


1 st). 


The second scenario for our COVID-19 projections, which we refer to as “COVID Loss 
Absenteeism”, draws on existing absenteeism literature. We first calculated an average effect 
size (in SD units) for each day missed of school by subject based on the effect sizes reported in 
Table | (e.g., an average -0.007 SDs per day in math and -0.004 SDs per day in reading). Next 
we converted these estimates into monthly losses on the RIT scale using NWEA’s subject- and 
grade-specific achievement norms (Thum & Kuhfeld, 2020), assuming there are approximately 
20 potential instructional days in a typical month and that students are absent during the entire 
school closure period. Given the majority of schools in the U.S. shut down around the week of 
March 15" (6.5 months into the school year), we used students’ projected achievement level at 
6.5 months (RIT, g) as the starting point for the projection and then assumed students lose 
ground from that point at that monthly rate calculated for each subject/grade. Given that students 
can only be absent while schools are still in session, we produced absenteeism projections only 


to the end of the school year (9.5 months). 
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RQI1. Possible Scenarios for Learning Gains during the 2019-20 School Year 


To display the possible scenarios for learning as a result of the school closures during the 
2019-20 school year, we produced a set of plots to compare these empirical- and literature-based 
projections to typical learning rates. The plots display students’ estimated learning rates across 
the 2019-20 school year and summer of 2020 based on the absenteeism and summer loss 
projections. In addition to the plots, we also reported the impact of school closures as a 
percentage of learning gains that students were expected to make relative to a typical school 
year. These percentages were calculated by estimating the total gains during the school year 
(subtracting the initial score on September 1°‘, 2019 from the projected score on June 15", 2020) 
under the two different COVID Loss assumptions and dividing those estimates by the total gains 


expected under typical growth. 
RQ2. Quantifying Variability in COVID-19 Impacts 


We do not expect that all students will be impacted by COVID-19 school closures 
equally. Prior summer learning loss research indicated that there is a considerable variability in 
students’ learning patterns over the summer (e.g., Atteberry & McEachin, 2019; Kuhfeld et al., 
2019), most of which cannot be explained by observed student and family characteristics (von 
Hippel et al., 2019; Kuhfeld, 2019; Borman, Benson, Overman, 2005). In addition to producing 
average estimates of learning rates during time out of school, we estimated variation in these 
learning rates across students. Specifically, we used the variance term of the within-school 
summer loss random effect (73;;) to examine the potential variability in COVID-19 impacts 
based on learning patterns during the summer of 2018. Based on the average monthly summer 
loss rate (Y399) and the standard deviation of the learning loss across students within the same 


school (Ts¢(3,3)), we calculated the monthly learning rates for students at the 25%, 50", and 75" 
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percentiles of the summer learning distribution. These estimates were then plotted to allow for an 
examination of the potential spread in fall 2020 RIT scores by grade/subject assuming students 
maintained the same rate of growth from school closure (March 15") to the start of the 2020-21 


school year. 


There are two potential limitations to this approach. First, while this approach allowed us 
to quantify variability in potential growth rates while students are out of school, it did not 
provide a direct estimate of the possible variability in test scores when students return to school 
following the COVID-19 school closures. Second, it ignored the correlation between gains made 
while in school and losses that occur out of school. Prior research has indicated school-year and 
summer learning are negatively correlated, with students who made the largest gains during the 
school year showing the biggest drops in the summer (e.g., Kuhfeld, 2019; von Hippel et al., 


2018). 


Therefore, we also used the empirical Bayes (EB) estimates of students’ learning rates 
from our models to project students’ achievement in fall 2020 under two scenarios. Under the 
first scenario, we used the EB estimates from the 2017-18 school year and the summer of 2018 to 
produce projected scores at the start of the 2018-19 school year. These projected fall scores were 
treated as what would be expected in fall 2020 under “business as usual”, had students completed 
the full 2019-20 school year and a typical summer break. The fall RIT scores are predicted using 
the following equation, in which 7 are parameter estimates from the model and fF are EB 


estimates of the random intercepts and slopes: 
Ri Ceynicateenat) = Yooo + toij + (F100 a Aiij) *9.5 + (P2090) * 9.57 (5) 


+ (P00 + P3i;) * 2.5, 
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In the second scenario, we assumed that COVID-19 increased the effects of summer loss 
by extending out of school time. In this case, projected fall scores were calculated for each 
student assuming a 6.5-month school year followed by a 5.5-month summer break, using the 


following equation: 
RITcovippangii = Pooo + foij + (F100 + Arij) * 6.5 + (200) * 6.57 (6) 
+ (P7300 + fsij) * 5-5 


Further details on the calculation of the projected scores under each scenario are provided 
in Appendix D. We then compared the distribution of scores under each condition to understand 
how much more variable the fall scores were under the COVID-19 Summer Slide assumption 


relative to a normal fall. 
RQ3. Estimating the Relationship Between Summer Loss and Next School Year’s Growth 


To guide planning to support student learning during this pandemic and school closures, 
it is important to understand not only the possible impact of school closures on student learning, 
but also whether students with large losses recover at similar or different rates than other 
students. To investigate this question, we examined the correlation among the learning rates 
during the summer of 2018 and in the 2018-19 school year. Specifically, we examined the level- 
2 random effect correlation matrix to understand the association between out of school learning 
rates and growth in the following school year. Though the empirical data are from a typical 
school year and summer, the results from this analysis can inform decision-making by serving as 


a proxy for student learning recovery post-COVID-19. 
Results 


RQI1. Possible Scenarios for Learning Gains during the 2019-20 School Year 
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Projected COVID-19 impacts on average academic growth trajectories are presented in 
Figure | for mathematics (Panel A) and reading (Panel B). In a typical year (shown as solid 
lines), average academic growth is not constant across the academic year (shown as the curved 
lines seen in some grades) and generally declines from the last day of school through the 
summer, with steeper declines in mathematics than in reading. The dashed line shows projected 
trajectories based on prior absenteeism literature (from COVID-19 school closure to the end of 
the 2019-20 school year), and dotted lines show projected trajectories under summer learning 
loss patterns (from COVID-19 school closure to start of the 2020-21 school year). Since the 
absenteeism estimates pertain to missing school while schools are still open, we did not extend 


the COVID Loss Absenteeism projections past June 15". 


Under both sets of projections, students’ learning gains are projected to be substantially 
lower at the end of the school year than under typical conditions. The COVID Loss Absenteeism 
projections for losses in learning are more dire than the COVID Loss Summer Slide projections, 
implying steeper drops while students are out of school across all grades and subjects. We also 
calculated the percentage of learning gains that students would be expected to have made relative 
to anormal year under each condition. Our results suggest that under the COVID Loss Summer 
Slide projections, students end the abbreviated 2019-20 school year with roughly 63-68% of the 
learning gains in reading relative to a typical school year (see Table D1 in the supplemental 
materials). However, in mathematics, students are likely to show much smaller gains, ending the 
school year with 37-50% of the average gains in a normal school year. For students moving from 
fifth to sixth grade, we expect under COVID Loss Summer Slide projections that students end 
the school year with only 19% of total mathematics gains. Under the COVID Loss Absenteeism 
projections, the story is even more dire, with students in sixth and seventh grade projected to end 


the school year with less than 30% of their typical learning gains in both math and reading. 
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RQ2. Quantifying Variability in COVID-19 Impacts 


Beyond average achievement, educators may be equally concerned about whether 
COVID-19 will result in greater variability in the academic skills that students bring with them 
when school resumes. In Figure 2, we display the variability in learning expected under the 
COVID Loss Summer Slide model from March 15" (when schools shut down) to September 1“ 
(when schools are expected to reopen). These estimates are based on variability seen during a 
typical summer, but with the duration of that summer extended. For parsimony, we only display 
Grades 4 and 6, but the model-based variability estimates for all grades/subjects are presented in 
Table D3 of the supplemental materials. The shaded areas display the spread in potential 
outcomes between students who were in the 25" percentile of summer learning loss (who 
showed steep declines) and those in the 75" percentile (who showed flat scores or even small 
gains during the summer). In mathematics, we see a fair amount of variability in learning rates, 
though the majority of students show losses over the extended closure and summer period. 
However, in reading, there is an even wider spread of potential outcomes, with students who are 
in the 75" percentile and above showing sizable learning gains during the summer. As seen in 
Table D3, approximately the upper half of the distribution (39-46% of students) are projected to 
show monthly gains in reading during the summer. Altogether, these plots show that extended 


time out of school may lead to more variability in achievement when students return in the fall. 


One limitation of the plots in Figure 2 is that they do not provide concrete evidence on 
the variability in fall achievement under COVID-19 relative to variability under a typical school 
year. Thus, in Figure 3 we display the spread of the projected fall 2020 test scores under 
“typical” conditions as well as the COVID Loss Summer Slide projections. The box plot shows 


the interquartile range (e.g., the 25", 50", and 75" percentiles) and the vertical lines extending 
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above and below the box stretch one and half times the interquartile range, with scores outside 
that range displayed as outliers (circles in the figure). The estimated means, SDs, and percentiles 
scores for each condition and grade/subject are reported in Table D3 in the supplemental 
materials. Across the board, students are projected to return in the fall with lower scores and 
more variability relative to a typical fall. In reading, the SDs of expected scores are expected to 
be up to 1.2 times the SDs expected in a typical fall. Thus, students will likely return not only 
with lower achievement (on average), but with a wider range of academic skills that may require 


teachers to further differentiate instruction. 
RQ3. Estimating the Relationship Between Summer Loss and Next School Year’s Growth 


Finally, to project whether larger COVID-19 learning losses would be associated with 
faster growth rates during the 2020-21 school year, we examined whether students who lost more 
ground during a typical summer showed slower rates of recovery during the subsequent typical 
school year. Correlations between students’ summer loss and linear growth during the 2018-19 
school year are presented in Tables C1 and C2 in the supplemental materials. In mathematics, 
student-level correlations ranged from -0.41 to -0.43, and in reading the correlations ranged from 
-0.45 to -0.46. These correlations imply that students who lost more ground during the summer 
of 2018 showed steeper growth during the following school year (2018-19) than students with 
less summer loss. Accordingly, this suggests that a student who lost ground during the summer 
does not necessarily continue to lose ground during the next school year; rather, they are likely to 


gain ground. 
Discussion 


Educators, policymakers, families, and students find themselves in uncharted territory 


during the COVID-19 crisis. School districts in particular are on the front lines to help ensure all 
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students have access to academic materials, instruction, and digital resources, among other basic 
needs such as food for students from low income backgrounds and support for students with 
disabilities, English learners, and students in temporary housing (Education Trust, 2020a). 
Despite these efforts, a majority of parents with children in K-12 schools are concerned that their 
children will fall behind academically due to the disruptions of COVID-19 school closures 
(Horowitz, 2020). In this study, we produced a set of possible scenarios for learning loss rates 
during the extended period when schools are physically closed and students are not receiving 
normal face-to-face instruction. These projections can help prepare educators and parents for the 
degree of variability in student achievement to expect when school resumes, including over the 


course of the upcoming school year. 


First, we show that students will likely (a) not have grown as much during the truncated 
2019-2020 academic year and (b) will likely lose more of those gains due to extended time out of 
school. Based on our projections, students will return in fall 2020 with approximately 63-68% of 
the learning gains in reading relative to a typical school year and with 37-50% of the learning 
gains in math. In some grades, students may come back close to a full year behind in math. 

While such projections may reinforce the worst fears of educators and parents, we should note 
that they do not factor in the home schooling and online instruction that students may currently 
be receiving. Therefore, they should be viewed as a likely upper bound for the potential negative 


effects on students’ learning. 


Second, we also examined variability in possible learning outcomes during the school 
closures and in the fall of 2020. We found that losing ground over the summer was not universal, 
with the top third of students in reading making gains during a typical summer. As a result of this 


variability, we project that the range of students’ academic achievement will be more spread out 
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in the fall of 2020 relative to a normal fall term, particularly in reading. In presenting these 
projections, we assume that the variability in typical summer loss can act as a proxy for the large 
variability in learning that is expected due to the widely differing home and school district 
conditions impacti learning during the school closure period. In all likelihood, differential access 
to parent and teacher supports for learning during the school closure months will produce 


variation larger than what typical summer break variability would imply. 


Finally, we show that, although our projections are dire, our models also suggest that 
students who lose the most while out of school tend to gain the most the following year (at least 
under typical summer loss conditions). Thus, there is hope that students most impacted by the 
additional average achievement losses under COVID-19 may also be the ones who rebound the 
most by the end of the 2020-21 academic school year. At the same time, one cannot be sure how 
financial uncertainty, health issues related to the virus, and psychological stresses may affect the 


association between summer loss and subsequent academic growth. 
Limitations of Our Projections 


While we provide two sets of projections in this study—one based on growth rates 
calculated from MAP Growth data and the other based on prior literature on student 
absenteeism—we acknowledge that it is impossible to accurately weigh the complex range of 
supports and challenges that students are facing during this period. The school closures caused 
by COVID-19 have additional aspects of trauma to students, loss of resources, and loss of 
opportunity to learn that go well beyond a traditional summer break for many families. In other 
words, families with financial resources, stable employment, and flexible work-from-home and 
childcare arrangements will likely weather this storm more easily than families who are renting 


their housing, working in low-paying fields that are hardest hit by the economic impacts, and 
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experiencing higher rates of food insecurity, family instability, and other shocks from this 


disruption. 


Given the uncertain impact of COVID-19, we have chosen not to make projections 
specific to inequalities by race/ethnicity, biological sex, and SES. Recent analyses of both ECLS- 
K and MAP Growth data have found little evidence that achievement gaps by race/ethnicity and 
SES widen during summer months (von Hippel & Hamrock, 2019; Kuhfeld, 2019). This is likely 
due to the fact that families of all income levels typically treat summer break as a vacation from 
math and reading, a time when “kids can be kids” (von Hippel, 2020). Were we to base estimates 
of COVID-19 impacts on racial/ethnic disparities in achievement and growth on these historical 
summer learning loss patterns, we would likely conclude that the COVID-19 pandemic is going 


to minimally impact long-standing inequalities in this country. 


However, there are many reasons to believe the COVID-19 impacts might be larger for 
children in poverty and children of color. There are higher rates of COVID-19 infections and 
deaths in the African American community (Bouie, 2020), and the economic downturn has been 
particularly damaging for African American and Hispanic parents, who are less likely to be able 
to work from home during the pandemic (Krogstad, Gonzalez-Barrera, & Noe-Bustamente 2020; 
Cerullo, 2020). Furthermore, the so-called “digital divide” in technology and internet access by 
race/ethnicity and socioeconomic status (Musu, 2018) likely contributes to greater inequalities 
during the COVID-19 pandemic than a typical summer. Given this evidence that the impacts of 
the COVID-19 school closures will have disproportionate impacts on our country’s most 
underserved communities in ways that historic summer data fails to capture, we chose not to 


produce projections based on pre-COVID-19 MAP Growth summer learning data for individual 
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subgroups. However, we believe it will be of great importance to study how existing inequalities 


have widened or been reshaped once schools have reopened. 


Furthermore, in calculating the projected impact of out-of-school time on learning in this 
study, we assumed that it is appropriate to linearly extrapolate learning loss from research on 
absenteeism and summer loss across the three months of school closure. Liu and colleagues 
(2019) found that additional absences had an approximately linear impact on student learning, 
though the number of absences assumed in this study (approximately 60 school days) far exceeds 
the average number of absences observed in their study. Furthermore, we have very little data 
about whether the summer months have a linear impact on students’ reading and mathematics 
skills. Campbell and Frey (1970) hypothesize that forgetting learned material may occur non- 
linearly, with rapid initial deceleration of knowledge followed by slower drop offs as time 
passes. However, we are unaware of any studies that have examined this phenomenon in the 
context of summer break. If the true effect of being out of school accelerates the longer students 
are out of school, we could be underestimating the impact on learning. But if summer loss simply 
reflects a process of forgetting and re-remembering that is not directly linked to the amount of 


time out of school, we could be greatly over-estimating the potential impacts on learning. 
Where Do We Go From Here? 


While we are not well-positioned to make recommendations for ways to remedy the 
learning loss that is likely occurring due to COVID-19, our results do provide takeaways that can 
inform how educators and leaders can prepare to support students upon return. First, we show 
that students may be substantially behind, especially in mathematics. Thus, teachers of different 


grade levels may wish to coordinate in order to determine where to start instruction. Educators 
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will also need to find ways to assess students early, either formally or informally, to understand 


exactly where students are academically. 


Second, students are likely to enter school with more variability in their academic skills 
than under normal circumstances. Prior research suggests greater heterogeneity in student 
achievement affects a classroom teacher’s ability to adapt instruction to meet the instructional 
needs of all students (Connor, Piasta, Fishman, Glasney, Schatschneider, Crowe, & Morrison, 
2009; Evertson, Sanford, & Emmer, 1981).Therefore, educators may need to consider ways to 
further differentiate instruction or provide opportunities for individualized learning. For a 
summary of related literature, one could turn to Peters, Rambo-Hernandez, Makel, Matthews, 


and Plucker (2017). 


Third, under typical schooling conditions, the students who lose the most during the 
summer tend to gain the most when back in school. Nonetheless, the ground that students have to 
make up during the 2020-21 academic year will probably be greater due to COVID-19. 
Therefore, educators may want to work with students to determine growth rates needed to catch 
up and set learning goals for the year that are ambitious but obtainable. These strategies might 
include establishing out-of-school learning supports during the 2020-21 school year for the 


students most affected by school closures. 


Finally, the effects of COVID-19 to which our study cannot speak may be ones most 
worthy of addressing. Districts are rushing to support educators who are attempting to teach 
academic content remotely while also caring for their students’ social emotional well-being. 
Prior research on students displaced by Hurricane Katrina indicated that students had difficulty 
concentrating and often manifested symptoms of depression in the months following the 


hurricane (Picou & Marshall, 2007). Understanding these impacts and how to best support 
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students’ social and emotional needs after this huge disruption of COVID-19 will be essential. 
Many students may face greater food insecurity, loss of family income, loss of family members 
to the coronavirus, and fear of catching the virus themselves (NAACP, 2020). While the scale of 
the COVID-19 school closures is novel, the inequalities in our school systems are unfortunately 
anything but new. Our models cannot account for the reality that the crisis is having an unequal 
impact on our most underserved communities. Nonetheless, we hope these analyses, which 
synthesize what we know from existing bodies of research, will inform tomorrow’s decision 


making. 
Conclusions 


These preliminary forecasts parallel many education leaders’ fears: missing school for a 
prolonged period will likely have major impacts on student achievement. Further, students will 
likely return in the fall of 2020 with greater variability in their academic skills. While we are 
unable to account for students’ exposure to virtual instruction while schools are closed, our 
learning loss projections imply that educators and policymakers will need to prepare for many 


students to be substantially behind academically when they return. 


Similar to the research that found students took nearly two full years to make up lost 
ground for the loss in instructional time due to Hurricane Katrina (Harris & Larsen, 2019), our 
COVID Loss projections provide new evidence on the scope of the long-term educational 
recovery efforts that will be required. We believe this study is one in a growing body of 
important work that leverages prior research to empower school leaders, policy makers, and 


researchers to make urgent evidence-informed post-COVID-19 recovery decisions. 
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Table 1 


Estimates of the Impact of Out-of-School Days on Standardized Test Scores Across Summer Loss, 
School Closure, and Absenteeism Literature 


Citation Location Grade level Math Effect ELA Effect 
Summer Loss 
Atteberry & McEachin (2019) National (NWEA) Ist grade -0.009 -0.010 
2nd grade -0.006 -0.006 
3rd grade -0.006 -0.005 
4th grade -0.005 -0.003 
5th grade -0.005 -0.003 
6th grade -0.003 -0.002 
7th grade -0.002 -0.001 
von Hippel, Workman, & Downey National (ECLS- Kindergarten 0.002 -0.001 
(2018) K:2011) Ist grade -0.001 -0.001 
Kuhfeld, Condron, & Downey National (NWEA) Kindergarten -0.005 -0.004 
(2019) Ist grade -0.007 -0.004 
3rd grade -0.006 -0.004 
4th grade -0.005 -0.003 
6th grade -0.004 -0.002 
7th grade -0.002 -0.001 
Absenteeism 
Liu, Lee, & Gershenson (2020) large urban CA school _ 6th-8th grade -0.008 -0.006 
district 
Gershenson, Jacknowitz, & ECLS-K + NC K-Ist grade —0.002 —0.002 
Brannegan (2017) NC public schools 3rd-5th grade -0.007 -0.004 
Aucejo & Romano (2016) NC public schools 3rd-Sth grade -0.006 -0.003 
School Closures due to Inclement Weather 
Hansen (2011) CO and MD public 8th grade (CO) -0.013 to -0.039 N/A 
schools 3rd grade (MD) -0.003 to -0.011 
(NS) 


5th grade (MD) -0.015 to -0.016 
8th grade (MD) -0.009 to -0.013 


Goodman (2014) MA public schools 3rd-8th + 10th -0.000 (NS) 0.003 (NS) 
grade 


Note. ECLS-K=Early Childhood Longitudinal Study, Kindergarten Cohort, CA=California, 
NC=North Carolina, CO=Colorado, MD=Maryland, MA=Massachusetts, NS=Not significant. All 
coefficients are reported as drops in standard deviation units on math and reading/English 
Language Arts assessments for each day of school missed. More details on each study are 
presented in Appendix Table A1. 
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Table 2 


Descriptive Statistics for the Sample 


Race/ethnicity 
Other 
Grade N. Schools N. Students White Black Asian Hispanic race Male % FRPL 
Mathematics 
3 12,816 986,862 0.45 0.18 0.04 0.18 0.14 0.51 0.51 
4 13,071 999,788 0.46 0.17 0.04 0.18 0.14 0.51 0.50 
5 14,146 1,029,363 0.47 0.17 0.05 0.18 0.13 0.51 0.50 
6 8,952 976,105 0.47 0.17 0.04 0.18 0.14 0.51 0.50 
7 7,040 937,054 0.47 0.16 0.04 0.18 0.13 0.51 0.50 
Full Sample 18,972 4,929,172 0.47 0.17 0.04 0.18 0.14 0.51 0.50 
Reading 
3 12,874 988,644 0.45 0.18 0.04 0.18 0.14 0.51 0.51 
4 13,066 997,088 0.47 0.18 0.04 0.18 0.14 0.51 0.51 
5 14,129 1,026,057 0.47 0.17 0.04 0.18 0.13 0.51 0.50 
6 8,943 970,524 0.47 0.17 0.04 0.18 0.14 0.51 0.50 
7 6,995 934,960 0.48 0.17 0.04 0.18 0.13 0.51 0.50 
Full Sample 18,958 4,917,273 0.47 0.17 0.04 0.18 0.14 0.51 0.50 


Note. N=Number, %FRPL=percentage of free or reduced priced lunch. Grade is the grade level students 


were in during the 2017-18 school year. 
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(A) Mathematics Projections 
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Figure 1. Mathematics and reading forecasts based on summer loss estimates and absenteeism 
literature. 
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(A) Mathematics Projections 
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Figure 2. Mathematics and reading forecasts for the 2019-20 school year accounting for the 
variability observed in typical summer loss patterns. 
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Figure 3. Projected fall 2020 score distributions under a typical fall (fall 2018) and COVID 
Loss Summer Slide conditions 
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Appendix A. Literature Review on Out-of-School Time Impacts 


In Tables Al-A3, we describe the studies included and excluded from our effect size 
table (Table 1 in the main paper) as well as the approach taken to convert the reported estimates 
into a single metric (SD loss per day of school missed). These articles were identified through a 
combination of database searches (e.g., Google Scholar), review of cited literature within recent 
studies, and inquiries of experts in each area. While we tried to capture key studies in each area, 
this review should not be considered a full representation of the work on each topic. In selecting 
studies for inclusion in Table 1, we prioritized recent studies where (a) the outcome was a math 
or reading test score and (b) the paper had estimates that could be converted into standard 
deviation (SD) units. For all studies that were included in Table 1, we present both the reported 
estimate (in the unit of the original reported coefficient) as well as the “Calculated” estimate 
(units of SD loss per day of school missed), which is consistent across all included studies. 


Table Al 


Review of Key Summer Learning Loss Studies 


Calc. Calc. Reported | Report 
Research Math ELA Math ELA 
Citation Location Design Grade Level Effect Effect Effect Effect | Description of Units 
Included in Literature Review Table 1 
Atteberry & National (NWEA | Seasonal Summer after Ist -0.009 -0.010 -0.19 -0.19 Scale of original estimate: total RIT point 
McEachin (2020) | - Ist to 8th grade) | growth Summer after 2nd -0.006 -0.006 -0.14 -0.10 drop across entire summer (reported in Tables 
model Summer after 3rd -0.006 -0.005 -0.13 -0.08 2 and 3) 
Summer after 4th -0.005 -0.003 -0.11 -0.06 
Summer after 5th -0.005 -0.003 -0.10 -0.06 Conversion: Divided estimate by 2020 spring 
Summer after 6th -0.003 -0.002 -0.06 -0.04 _| SDs (by grade/subject) and then converted into 
instructional days (assuming approx. 50 
Summer after 7th weet veers oe is weekdays adie i Sen 
von Hippel, National (ECLS- | Seasonal Summer after K 0.002 -0.001 0.03 -0.01 Scale of original estimate: Monthly SD units 
Workman & K:2011) growth Summer after 1st -0.001 -0.001 -0.02 -0.02 (reported in Table 4) 
Downey (2018) model 
Conversion: Divided by 20 weekdays per 
month to get SD per day 
Kuhfeld, National (NWEA, | Seasonal Summer after K -0.005 -0.004 -1.19 -1.00 Scale of original estimate: RIT point drop per 
Condron, & K-8) growth Summer after Ist -0.007 -0.004 -1.89 -1.06 summer month (reported in Table 2) 
Downey (2019) model Summer after 3rd -0.006 -0.004 -1.72 -1.14 
Summer after 4th -0.005 -0.003 -1.58 -0.88 Conversion: Divided estimate by 2020 spring 
Summer after 6th -0.004 -0.002 -1.44 -0.75 SDs (by grade/subject) and then divided by 20 
Summer after 7th -0.002 -0.001 -0.85 -0.41 weekdays per month to get SD per day 
Excluded from Literature Review Table 1 
Cooper, Nye, 13 different Meta- Ist-9th N/A N/A -0.14 -0.05 Reported estimates are in SD units across the 
Charlton, studies analysis whole summer (pg. 253). This study was 
Lindsay, & excluded due to measurement issues (described 
Greathouse by von Hippel & Hamrock, 2019) in the 
(1996) studies reviewed. 
Quinn, Cooc, National (ECLS- | Seasonal K-2nd N/A N/A N/A N/A This study only compares race/SES differences 
McIntyre, & K:2011) growth in summer loss and does not provide overall 
Gomez, (2016) model summer drop estimates, so we did not include 
the results from this study in Table 1. 
Kuhfeld (2019) National Projected | K-7th N/A N/A 70-78% 62% to | This paper reports the percentage of students 
(NWEA) test scores lost 73% who lost ground during the summer, which can 
ground not be translated into SD units. 
von Hippel & National (ECLS- | Seasonal K-8th N/A N/A N/A N/A This study only compares race/SES differences 
Hamrock (2019) | K:2011 + growth in summer loss and does not provide overall 
NWEA) and model summer drop estimates, so we did not include 
Baltimore schools the results from this study in Table 1. 


Table A2 


Review of Key School Closures Studies 


Calc. Math Cale. ELA 
Research Effect Effect Reported Math Reported ELA 
Citation Location | Design Grade Level Effect Effect Notes 
Included in Literature Review Table 1 
Hansen CO; MD | Two-sample | 8th grade -0.013 to - N/A .013-.039 (NS) N/A Scale of original estimate: SD units for every 
(2011) least (CO) 0.039 absence (reported in Table 6 on pg 36) 
squares 3rd grade -0.003 to - .003-.011 (NS) 
(MD) 0.011 (NS) Conversion: None necessary 
5th grade -0.015 to - .015-.016 
(MD) 0.016 
8th grade -0.009 to - .009- .013 
(MD) 0.013 
Goodman MA Instrumental | 3rd-8th + 10th | -0.000 (NS) | 0.003 (NS) | -0.000 (NS) 0.003 (NS) Scale of original estimate: SD units for every 
(2014) variables grade day closed (reported as maximum plausible 
effect size), Column (IV) Table 6 on pg 35 
Conversion: None necessary 
Excluded from Literature Review Table 1 
Sacerdote LA/TX Difference- | students N/A N/A initial decline of | initial decline The comparison reported in this study is by 
(2012) in- impacted by 0.1 SD, but of 0.1 SD, then | "evacuee" status. The reported estimate is drop 
Difference Hurricane gained back gained back one year later on standardized test scores, 
Katrina within 3 years which we could not convert into the SD/day 
metric of the other studies so it was excluded 
from Table 1. 
students N/A N/A Initial decline of | initial decline The comparison reported in this study is by 
impacted by 0.08 SD, then of 0.06 SD, and | "evacuee" status. The reported estimate is drop 
Hurricane Rita gained slightly not gained back | one year later on standardized test scores, 
by 2009 which we could not convert into the SD/day 
metric of the other studies so it was excluded 
from Table 1. 
Ward, MS Means of 3rd to 8th N/A N/A 5-7 points 4-7 points The comparison reported in this study was 
Shelley, scale scores | grade students who were and were not "displaced" by 
Kaase, & Hurricane Katrina. We are unable to covert the 
Pane (2008) mean scale score differences into SD units so it 
was excluded from Table 1. 


Marcotte MD Regression | 3rd grade N/A N/A -1.20% -0.78% The reported metric was change in % of 
(2007) students getting 'satisfactory' per SD increase in 
snow accumulation. We are unable to covert 
the mean scale score differences into SD units 
5th grade N/A N/A -0.93% not sig so it was excluded from Table 1. 
8th grade N/A N/A -0.94% not sig 
Marcotte & | MD Regression | 3rd grade N/A N/A -0.53% -0.51% The reported metric was change in % of 
Hemelt 5th grade N/A N/A smaller than 3rd smaller than 3rd students getting ‘satisfactory’ per day lost. We 
(2007) grade grade are unable to covert the mean scale score 
differences into SD units so it was excluded 
8th grade N/A N/A smaller than 3rd__| half 3rd grade from Table 1. 


grade 


Table A3 


Review of Key Absenteeism Studies 


Calc. Calc. Reported Report 
Research Grade Math ELA Math ELA 
Citation Location Design Level Effect Effect Effect Effect Notes 
Included in Literature Review Table 
Liu, Lee, & large urban Lagged-score middle -0.008 -0.006 -0.077 -0.057 Scale of original estimate: SD units for every 10 spring 
Gershenson CA school VAM and school absences. Math results are taken from column (5) of 
(2020) district between-subject Table 4 and ELA results are results are taken from 
differences high -0.009 -0.008 -0.085 -0.075 | column (5) of Table A2. 
school 
Conversion: Divided by 10 to get SD units per day 
Gershenson, es ies noe oe ge? Ge? oe Scale of original estimate: SD units for every absence. 
Jacknowitz, & NC public pes 3rd-Sth -0.007 -0.004 -0.007 -0.004 Results are taken from Table 4. 
Brannegan schools 
(2017) Conversion: None necessary 
NC public Student/School 3rd — Sth -0.006 -0.003 -0.0055 -0.0029 ae . : 
écheols fied Siisets Scale of original estimate: SD units for every absence. 
Aucejo & mdela Results taken from columns (5) from Table 3. 
Romano (2016) instraniedéal 
arable Conversion: None necessary. 
Excluded from Lit Review Table 
Gottfried & small urban School and 3rd-Sth N/A N/A -0.07 -0.03 The reported metric is change in SD for one spring 
Kirskey (2017) | CA district classroom fixed absence (pg. 124). 
effects. 
Gottfried Philadelphia | Family-FE 2nd-4th N/A N/A -0.10 -0.08 We are reporting the effect sizes from pg. 172 from the 
(2011) school estimates grade family fixed effects models. These results are from 
district comparing siblings in the same family. 
Gottfried Philadelphia | Lagged test 2nd-4th N/A N/A -0.099 -0.054 We are reporting the VAM estimates (Column 8) from 
(2009) school score VAM grade Table 4 and 7. 
district 
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Appendix B. Comparison of NWEA and US Public Schools 


NWEA schools. We defined a NWEA school as one that tests at least 10 students in 
Grades 3-8 in math or reading in either 2017-18 or 2018-19. The final sample included 18,958 
NWEA schools. 


Population of Public Schools. We downloaded the 2017-18 Public 
Elementary/Secondary School Universe Survey data file from 
https://nces.ed.gov/ccd/pubschuniv.asp. We define the population of interest as the set of US 
operational (e.g., school status variable SY STATUS does not indicate the school is closed or 
yet to be opened) public schools in the 50 states and District of Columbia serving students in 
grades 3-8 (based on the minimum (GSLO) and maximum (GSHI) grades offered at the school). 
In 2017-18, the population of interest consists of 72,075 schools. The NCES school 
characteristics included in our comparison include enrollment by grade, percentage of students 
receiving free or reduced-price lunch (TOTFRL divided by school enrollment), and percentages 
of the students in the school who were Hispanic, Black, and White, and Asian (HI, BL, WH and 
AS) divided by total enrollment, and urbanicity (NCES’ LOCALE codes, collapsed into City, 
Suburb, Town, and Rural). 


Comparison of Sample and Population. Table B1 presents the comparison between the 
NWEA schools and population of US public schools serving students in Grades 3-8 based on 
2017-18 CCD information. The NWEA and population of schools match closely on Percent 
FRPL and urbanicity, but there were small differences in the percentage of students enrolled of 
different racial/ethnic groups. Specifically, the NWEA sample of schools has a slightly higher 
percentage of Black students on average (17% vs. 15% overall) and lower percentage of 
Hispanic students (20% vs. 24% overall). 


Table Bl 


Characteristics of the NWEA Sample of Schools Relative to the US Population of Public Schools 
Population of US Public 


NWEA Sample of Schools Schools Serving Grades 3-8 


N M SD N M SD 

3rd grade 13,699 71.51 42.55 53,430 70.87 44.72 
4th grade 13,621 73.03 45.54 53,180 72.54 47.21 
5th grade 13,220 75.83 54.75 51,881 74.69 55.75 
6th grade 9,006 104.08 107.37 37,688 101.47 110.08 
7th grade 7,452 123.14 122.29 31,995 117.88 129.00 
8th grade 7,344 124.27 124.38 31,770 118.47 130.32 
Percent FRPL 18,479 0.50 0.30 72,062 0.51 0.31 
Percent Hispanic 18,480 0.20 0.24 72,063 0.24 0.27 
Percent Black 18,480 0.17 0.25 72,063 0.15 0.23 
Percent White 18,480 0.53 0.33 72,063 0.51 0.33 
Percent Asian 18,480 0.04 0.07 72,063 0.04 0.09 
City 18,483 0.29 0.45 72,075 0.28 0.45 
Suburb 18,483 0.33 0.47 72,075 0.33 0.47 
Town 18,483 0.11 0.32 72,075 0.11 0.32 
Rural 18,483 0.26 0.44 72,075 0.28 0.45 


Appendix C. Parameter Estimates from Multilevel Growth Models 


Tables C1 and C2 display the random and fixed effects from the quadratic growth models 
for mathematics and reading, respectively. The right half of the tables show the school- and 
student-level correlations among the random effects. Tables C3 and C4 display the random and 
fixed effects from the linear growth models for mathematics and reading, respectively. These 
models were estimated to confirm the findings that were observed with the quadratic growth 
models. 


Table Cl 


Multilevel Quadratic Growth Model Parameter Estimates for Mathematics 


Subject Grade Growth Term 


Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
Math 
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NIANYANANIANIANTDADAAYVDA|AQIN AAA HIP HHP HP BP HPILW W W W W 


Intercept 

Linear Growth - Year 1 
Quadratic Growth - Year 1 
Summer Drop 

Linear Growth - Year 2 
Quadratic Growth - Year 2 
Intercept 

Linear Growth - Year 1 
Quadratic Growth - Year 1 
Summer Drop 

Linear Growth - Year 2 
Quadratic Growth - Year 2 
Intercept 

Linear Growth - Year 1 
Quadratic Growth - Year 1 
Summer Drop 

Linear Growth - Year 2 
Quadratic Growth - Year 2 
Intercept 

Linear Growth - Year 1 
Quadratic Growth - Year 1 
Summer Drop 

Linear Growth - Year 2 
Quadratic Growth - Year 2 
Intercept 

Linear Growth - Year 1 
Quadratic Growth - Year 1 
Summer Drop 

Linear Growth - Year 2 
Quadratic Growth - Year 2 


Fixed Effect 
186.24 (0.06)*** 
2.23 (0.01)*** 
-0.06 (0.00)*** 
-1.35 (0.01)*** 
1.49 (0.01)*** 
-0.01 (0.00)*** 
198.45 (0.06)*** 
1.47 (0.01)*** 
0.00 (0.00)*** 
-1.45 (0.01)*** 
1.28 (0.01)*** 
-0.01 (0.00)*** 
207.66 (0.07)*** 
1.34 (0.01)*** 
-0.01 (0.00)*** 
-2.51 (0.01)*** 
1.11 (0.01)*** 
-0.01 (0.00)*** 
211.97 (0.09)*** 
1.20 (0.01)*** 
-0.02 (0.00)*** 
-1.04 (0.01)*** 
0.87 (0.01)*** 
-0.01 (0.00)*** 
218.30 (0.11)*** 
0.96 (0.01)*** 
-0.01 (0.00)*** 
-0.79 (0.02)*** 
0.90 (0.01)*** 
-0.02 (0.00)*** 


6.24 
0.33 


0.90 
0.32 


6.72 
0.34 


0.90 
0.34 


7.66 
0.36 


1.12 
0.32 


8.20 
0.36 


0.91 
0.31 


8.97 
0.35 


0.97 
0.34 


SchoolSD Student SD ICC 


Random Effects 
12.01 0.21 
0.78 0.15 
1.98 0.17 
0.76 0.15 
12.40 0.23 
0.77 0.16 
2.01 0.17 
0.79 0.16 
13.62 0.24 
0.81 0.17 
2.30 0.19 
0.76 0.15 
14.07 0.25 
0.80 0.17 
217 0.15 
0.80 0.13 
15.64 0.25 
0.83 0.15 
2.29 0.15 
0.85 0.14 
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Int. 
1.00 
0.00 


0.08 
0.17 


1.00 
0.20 


-0.01 
0.23 


1.00 
0.25 


-0.26 
0.19 


1.00 
0.17 


-0.05 
0.18 


1.00 
0.13 


-0.04 
0.14 


School-level Correlation 


Lin. - Yl Sum. 
1.00 

-0.71 1.00 
0.26 -0.31 
1.00 

-0.68 1.00 
0.32 -0.35 
1.00 

-0.72 1.00 
0.22 -0.29 
1.00 

-0.77 1.00 
0.34 -0.36 
1.00 

-0.75 1.00 
0.47 -0.55 


Lin. - Y2 


1.00 


1.00 


1.00 


1.00 


1.00 


Int. 
1.00 
-0.20 


0.11 
0.08 


1.00 
-0.06 


0.07 
0.14 


1.00 
0.00 


-0.19 
0.15 


1.00 
-0.02 


0.02 
0.11 


1.00 
-0.06 


0.00 
0.13 


Student-level Correlation 


Lin. - Yl Sum. 
1.00 

-0.57 1.00 
0.06 -0.42 
1.00 

-0.57 1.00 
0.08 -0.41 
1.00 

-0.65 1.00 
0.08 -0.41 
1.00 

-0.58 1.00 
0.07 -0.42 
1.00 

-0.60 1.00 
0.07 -0.43 


Lin. - Y2 


1.00 


1.00 


1.00 


1.00 


1.00 


Table C2 


Multilevel Quadratic Growth Model Parameter Estimates for Reading 


Random Effects School-level Correlation 
Subject Grade Growth Term Fixed Effect SchoolSD Student SD ICC Int. Lin. - Y1 Sum. Lin. - Y2 
Reading 3 Intercept 184.80 (0.07) *** 7.33 15.56 0.18 1.00 
Reading 3 Linear Growth - Year 1 2.33 (0.01)*** 0.32 1.03 0.09 -0.21 1.00 
Reading 3 Quadratic Growth - Year 1 -0.10 (0.00)*** 
Reading 3 Summer Drop -0.75 (0.01)*** 0.92 2:72 0.10 0.08 -0.73 1.00 
Reading 3 Linear Growth - Year 2 1.71 (0.01)*** 0.28 0.95 0.08 -0.19 0.25 -0.38 1.00 
Readng 3 Quadratic Growth - Year 2 -0.08 (0.00)*** 
Readng 4 Intercept 195.28 (0.07) *** 7.35 15.18 0.19 1.00 
Readng 4 Linear Growth - Year | 1.77 (0.01) *** 0.29 0.96 0.08 -0.23 1.00 
Reading 4 Quadratic Growth - Year 1 — -0.08 (0.00)*** 
Reading 4 Summer Drop -0.58 (0.01)*** 0.85 2.61 0.10 0.05 -0.71 1.00 
Readng 4 Linear Growth - Year 2 1.44 (0.01)*** 0.27 0.90 0.08 -0.21 0.27 -0.40 1.00 
Reading 4 Quadratic Growth - Year 2 —_ -0.07 (0.00)*** 
Reading 5 Intercept 202.58 (0.07)*** 7.44 14.87 0.20 1.00 
Reading 5 Linear Growth - Year 1 1.51 (0.01)*** 0.27 0.93 0.08 -0.27 1.00 
Reading 5 Quadratic Growth - Year 1 — -0.08 (0.00)*** 
Reading 5 Summer Drop -0.53 (0.01)*** 0.87 2.63 0.10 0.07 -0.67 1.00 
Reading 5 Linear Growth - Year 2 1.02 (0.01)*** 0.27 0.93 0.08 -0.18 0.20 -0.31 1.00 
Reading 5 Quadratic Growth - Year 2 -0.05 (0.00)*** 
Reading 6 Intercept 207.89 (0.09) *** 7.66 14.90 0.21 1.00 
Reading 6 Linear Growth - Year | 1.11 (0.01)*** 0.30 0.96 0.09 -0.22 1.00 
Reading 6 Quadratic Growth - Year 1 -0.05 (0.00)*** 
Reading 6 Summer Drop -0.44 (0.01)*** 0.88 2.71 0.10 0.06 -0.73 1.00 
Reading 6 Linear Growth - Year 2 0.83 (0.01)*** 0.27 0.95 0.08 -0.15 0.30 -0.41 1.00 
Reading 6 Quadratic Growth - Year 2 —_ -0.04 (0.00)*** 
Reading 7 Intercept 212.20 (0.10)*** 7.67 15.04 0.21 1.00 
Reading 7 Linear Growth - Year 1 0.90 (0.01)*** 0.31 0.99 0.09 -0.22 1.00 
Readng 7 Quadratic Growth- Year 1 -0.04 (0.00)*** 
Reading 7 Summer Drop -0.27 (0.02)*** 0.93 2.78 0.10 0.03 -0.73 1.00 
Reading 7 Linear Growth - Year 2 0.86 (0.01)*** 0.29 0.97 0.08 -0.14 0.38 -0.54 1.00 
Reading 7 Quadratic Growth - Year 2 —_ -0.05 (0.00)*** 
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Int. 
1.00 
-0.39 


0.15 
-0.17 


1.00 
-0.38 


0.15 
-0.15 


1.00 
-0.37 


0.14 
-0.11 


1.00 
-0.34 


0.12 
-0.08 


1.00 
-0.32 


0.09 
-0.07 


Student-level Correlation 


Lin. - Yl Sum. 
1.00 

-0.58 1.00 
0.05 -0.45 
1.00 

-0.59 1.00 
0.04 -0.46 
1.00 

-0.57 1.00 
0.02 -0.46 
1.00 

-0.58 1.00 
0.02 -0.46 
1.00 

-0.59 1.00 
0.01 -0.46 


Lin. - Y2 


1.00 


1.00 


1.00 


1.00 


1.00 


Table C3 


Multilevel Linear Growth Model Parameter Estimates for Mathematics 


Random Effects School-level Correlation Student-level Correlation 
School Student 
Subject Grade Growth Term Fixed Effect SD SD ICC Int. Lin.- Y1Sum. Lin. - Y2 Int. Lin.- Y1 Sum. Lin. - Y 
Math 3 Intercept 186.91 (0.06)*** 6.21 12.01 0.21 1.00 1.00 
Math 3 Linear Growth- Year 1 1.69 (0.00)*** 0.33 0.78 0.15 0.02 ~~ 1.00 -0.20 1.00 
Math 3 Summer Drop -1.62 (0.01)*** 0.93 1.99 0.18 0.07 = -0.68 1.00 0.11 = -0.57 1.00 
Math 3 Linear Growth - Year2 1.42 (0.00)*** 0.32 0.76 0.15 0.18 0.26 -0.30 1.00 0.08 0.06 -0.42 1.00 
Math 4 Intercept 198.50 (0.06)*** 6.71 12.40 0.23 1.00 1.00 
Math 4 Linear Growth- Year 1 1.42 (0.00)*** 0.34 0.77 0.16 0.20 1.00 -0.06 1.00 
Math 4 Summer Drop -1.45 (0.01)*** 0.90 2.01 0.17 -0.01  -0.67 1.00 0.07  -0.57 1.00 
Math 4 Linear Growth - Year 2 1.23 (0.00)*** 0.34 0.79 0.16 0.23 0.32 -0.35. 1.00 0.14 0.08 -0.41 1.00 
Math 5 Intercept 207.75 (0.07)*** 7.66 13.62 0.24 1.00 1.00 
Math 5 Linear Growth- Year 1 1.26 (0.00)*** 0.36 0.81 0.17. 0.25 1.00 0.00 1.00 
Math 5 Summer Drop -2.51 (0.01)*** 1.12 2.30 0.19 -0.27  -0.72 1.00 -0.19  -0.65 1.00 
Math 5 Linear Growth - Year2 0.99 (0.00)*** 0.32 0.76 0.15 0.20 0.22 -0.29 1.00 0.15 0.08 -0.41 = 1.00 
Math 6 Intercept 212.16 (0.09)*** 8.19 14.07 0.25 1.00 1.00 
Math 6 Linear Growth- Year 1 1.04 (0.00)*** 0.36 0.80 0.17 0.18 1.00 -0.02 1.00 
Math 6 Summer Drop -1.10 (0.01)*** 0.91 2.17 0.15 -0.06 -0.76 1.00 0.02 -0.58 1.00 
Math 6 Linear Growth - Year 2 —_ 0.79 (0.00)*** 0.31 0.80 0.13 0.18 0.34 -0.35. 1.00 0.11 0.07 -0.42 1.00 
Math 7 Intercept 218.46 (0.11)*** 8.97 15.64 0.25 1.00 1.00 
Math 7 Linear Growth- Year 1 0.84 (0.00)*** 0.34 0.83 0.15 0.13 1.00 -0.06 1.00 
Math 7 Summer Drop -0.77 (0.01)*** 0.95 2.29 0.15 -0.05  -0.74 1.00 0.00  -0.60 1.00 
Math 7 Linear Growth - Year 2 0.69 (0.00)*** 0.34 0.85 0.14 0.14 0.45 -0.54 1.00 0.13 0.07 -0.43 1.00 
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Table C4 


Multilevel Linear Growth Model Parameter Estimates for Reading 


Subject 

Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 
Reading 


Grade 


NIAYNANNNDADAAD AINA AN NSP HP HP HPI] wW W W Ww 


Growth Term 


Intercept 


Linear Growth - Year | 


Summer Drop 


Linear Growth - Year 2 


Intercept 


Linear Growth - Year | 


Summer Drop 


Linear Growth - Year 2 


Intercept 


Linear Growth - Year | 


Summer Drop 


Linear Growth - Year 2 


Intercept 


Linear Growth - Year | 


Summer Drop 


Linear Growth - Year 2 


Intercept 


Linear Growth - Year | 


Summer Drop 


Linear Growth - Year 2 


Fixed Effect 
185.95 (0.07)*** 
1.39 (0.00)*** 
-0.99 (0.01)*** 
0.99 (0.00)*** 
196.18 (0.07)*** 
1.02 (0.00)*** 
-0.74 (0.01)*** 
0.79 (0.00)*** 
203.39 (0.07)*** 
0.83 (0.00)*** 
-0.76 (0.01)*** 
0.59 (0.00)*** 
208.44 (0.09)*** 
0.65 (0.00)*** 
-0.56 (0.01)*** 
0.48 (0.00)*** 
212.66 (0.10)*** 
0.53 (0.00)*** 
-0.30 (0.01)*** 
0.41 (0.00)*** 


Random Effects 

School Student 

SD SD ICC 
7.31 15.55 0.18 
0.35 1.03 0.10 
1.00 2.72 0.12 
0.29 0.95 0.09 
7.34 15.17 0.19 
0.31 0.97 0.09 
0.91 2.61 0.11 
0.28 0.91 0.09 
7.43 14.86 0.20 
0.29 0.93 0.09 
0.93 2.63 0.11 
0.28 0.93 0.08 
7.64 14.90 0.21 
0.31 0.96 0.10 
0.91 2.71 0.10 
0.27 0.95 0.08 
7.66 15.03 0.21 
0.31 0.99 0.09 
0.94 2.78 0.10 
0.29 0.97 0.08 
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School-level Correlation 


Int. 
1.00 
-0.19 
0.06 
-0.16 
1.00 
-0.22 
0.05 
-0.18 
1.00 
-0.26 
0.06 
-0.16 
1.00 
-0.22 
0.05 
-0.13 
1.00 
-0.22 
0.02 
-0.12 


Lin. - 
Y1 


1.00 
-0.73 
0.28 


1.00 
-0.72 
0.30 


1.00 
-0.69 
0.20 


1.00 
-0.73 
0.28 


1.00 
-0.72 
0.36 


Sum. 


1.00 
-0.41 


1.00 
-0.42 


1.00 
-0.33 


1.00 
-0.40 


1.00 
-0.53 


Lin. - 
Y2 


1.00 


1.00 


1.00 


1.00 


1.00 


Student-level Correlation 


Int. 
1.00 
-0.39 
0.15 
-0.16 
1.00 
-0.38 
0.15 
-0.15 
1.00 
-0.37 
0.14 
-0.11 
1.00 
-0.34 
0.12 
-0.08 
1.00 
-0.31 
0.09 
-0.06 


Lin. - Y1 Sum. 
1.00 
-0.58 1.00 
0.05 -0.45 
1.00 
-0.59 1.00 
0.04 -0.46 
1.00 
-0.57 1.00 
0.02 -0.46 
1.00 
-0.58 1.00 
0.02 -0.46 
1.00 
-0.59 1.00 
0.01 -0.46 


Lin. - 
Y2 


1.00 


1.00 


1.00 


1.00 


1.00 


Appendix D. Supplemental Results 


Table D1 contains (a) the predicted monthly drop in RIT scores from the absenteeism 
literature, (b) estimates of typical fall-spring growth across 9.5 months (based on the linear 
Yio0 and quadratic 72 99growth parameters) and summer loss (f/3o9,) based on parameters from 
quadratic growth model, (c) projected gains by the end of the school year under COVID Loss 
Absenteeism and COVID Loss Summer Slide (assuming students were in school for 6.5 months 
followed by three months of out of school time), and (d) percentage of learning gains made 
relative to a typical school year under the two projections. The COVID Loss Absenteeism rate 
was calculated by averaging the effect size estimates from the absenteeism studies in Table 1 of 
the paper (separately for mathematics and Reading/ELA) and converting those SD drops into 
RIT units using the spring RIT SDs per/grade subject in NWEA’s 2020 Norms (see Thum & 
Kuhfeld, 2020). With the exception of 5 grade mathematics, the COVID Loss Absenteeism 
estimates implied larger RIT drops per month than the COVID Loss Summer Slide projections. 


Projected gains by the end of the school year under typical growth, COVID Loss 
Absenteeism, and COVID Loss Summer Slide assumptions were calculated as follows. 
Assuming that students were learning at a typical rate 6.5 months out of a standard 9.5-month 
followed by three months of learning lost at each projected rate, the estimate gain would be 


Galnrypicatg = (Pr00) * 9.5 + (200) * 9-57 


GaiMapstent,g = (Vr00) * 6-5 + P200) * 6.5 + (Absent) * 3 


GalNsummer,g = (Pr00) * 6.5 + (F200) * 6.57 + (F300) * 3, 


where the Absent is the monthly absenteeism rate reported in Table D1 and the parameters 
estimates are presented in Tables C1 and C2. The percentage of learning gains made relative to a 
typical year is calculated by dividing the projected gains under each scenario (Gain Abstent,g and 
GalNsummer,g) by the typical gains (Gainrypicaig) estimated in each grade/subject. The 
percentages of learning gains made under the two scenarios (final columns of Table D1) reveal 
that students may be expected to show large losses, particularly in math, due to the COVID 
school closures. 


Table D2 presents the summer loss parameter estimates (e.g., the parameter estimate 
¥300 and random effect SD VTse(3,3)) from the multilevel growth models as well as the monthly 
learning rates for students at the 25", 50", and 75" percentiles of the summer learning 
distribution across grades/subjects. Additionally, we report the percentile in the summer learning 
distribution at which students show monthly gains rather than losses. In both mathematics and 
reading, there is a large amount of variability in summer learning rates. In mathematics, students 
in the top 20-30% of the distribution (depending on the grade) actually show monthly gains 
rather than losses. In reading, approximately the upper half of the distribution (39-46% of 
students) show gains during the summer. Based on these findings, it is clear that summer loss is 
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common but far from universal, and provide some evidence that we could expect that the 
COVID-19 extended school closures may not be associated with academic loss for all students. 


As a second part of Research Question 2, we compared projected fall scores under two 
different scenarios based on the empirical Bayes (EB) estimates from the models fit to the 2017- 
18 and 2018-19 MAP Growth data. The first scenario assumes “typical” fall scores assuming 
student i in grade g within school 7 completed the prior school year and had a standard summer 
break: 


RIT pypical_Falt,gij = Vooo + Poij a (F100 + Arij) *9.5 + (P20) * 9.5% + (P00 a5 P3i;) * 2.5. 


In the second scenario, we produce COVID-19 projected fall assuming students were out of 
school during the last three months of the 2019-20 school year: 


RITcovip Fatt gij = Yooo + foi + (P00 a Aiij) * 6.5 + (200) * 6.57 + (7300 + P3i;) * 5.5. 


For each grade/subject, we calculated projected fall scores under each scenario for all of the 
students in the analysis. Table D3 displays the predicted means, SDs, and percentile scores 
(based on NWEA’s 2020 Norms) under two scenarios. We observed that the test scores under the 
COVID-19 projections are more variable, with SDs that are slightly larger in mathematics and 
close to 16-20% larger in reading as compared with SDs in a typical fall. Also, while under 
normal conditions this sample of students’ projected scores is close to the national norms on 
average (e.g., near 50" percentile in their fall scores), under the COVID-19 Summer Slide 
projections these students would be considered well-below average in the fall based on NWEA’S 
grade/subject-specific norms. 
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Table D1 
Projected Gains Retained at the End of the 2019-20 School Year 


2017-18 Results (based on 


Projected Gains by the End of 


Percentage of Learning Gains 
Made Relative to a Typical 


Quadratic Growth Model) School Year School Year 
Absenteeism Summer 

Drops per Fall Fall-Spring Drop Per COVID Loss COVID Loss COVID Loss COVID Loss 

Grade Subject Month Score Growth Month Absenteeism Summer Slide Absenteeism Summer Slide 
3 Mathematics  -1.62 186.24 15.77 -1.35 7.09 7.91 45% 50% 
4 Mathematics -1.79 198.45 = 13.97 -1.45 4.19 5.21 30% 37% 
5 Mathematics  -1.92 207.66 =—11.83 -2.01 2.53 2.26 21% 19% 
6 Mathematics  -2.01 211.97 9.60 -1.04 0.93 3.84 10% 40% 
7 Mathematics _-2.14 218.30 8.22 -0.79 0.12 3.45 1% 42% 
3 Reading -1.14 184.80 13.11 -0.75 7.50 8.67 57% 66% 
4 Reading -1.14 195.28 9.60 -0.58 4.70 6.39 49% 67% 
5 Reading -1.12 202.58 7.13 -0.53 3.08 4.85 43% 68% 
6 Reading -1.12 207.89 6.03 -0.44 1.74 3.78 29% 63% 
7 Reading -1.15 212.20 4.94 -0.27 0.72 3.35 15% 68% 


Note. The absenteeism rate (reported in RIT points per month) is a transformation of the SD results seen in existing literature whereas 


the 2017-18 results presented are model-based estimates based on the quadratic model results displayed in Tables C1 and C2. 
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Table D2 


Variability in Summer Learning Loss Estimates During Summer of 2018 


Monthly Learning Loss 

Summer at Different Points in the 
Drop Summer Perc. at Which Distribution 
Fixed Drop Students Show 25th 50th 75th 

Grade Subject Effect SD Gains Perc. Perc. Perc. 

3 Mathematics -1.35 1.98 715% -2.68 -1.35  -0.01 

4 Mathematics -1.45 2.01 76% -2.81 -1.45 -0.09 

5 Mathematics -2.01 2.30 81% -3.55 -2.01 -0.45 

6 Mathematics -1.04 2.17 68% -2.51 -1.04 0.42 

7 Mathematics  -0.79 2.29 64% -2.33 -0.79 0.75 

3 Reading -0.75 Di fs 61% -2.58 -0.75 1.09 

4 Reading -0.58 2.61 59% -2.34 -0.58 1.18 

) Reading -0.53 2.63 58% -2.30 -0.53 1.24 

6 Reading -0.44 2.11 56% -2.26 -0.44 1.39 

7 Reading -0.27 2.78 54% -2.15  -0.27 1.60 


Note. Perc=Percentile. Reported estimates are monthly gains/losses in RIT points during the 
summer months. The reported percentile is the estimated percentile in which students are 
showing positive monthly growth rates in either reading or mathematics during the summer. 


Table D3 
Fall 2020 Score Projections Under “Typical” and COVID-19 Conditions 
COVID-19 (Summer Slide) 


"Typical" Fall Scores Projected Fall Scores 
Grade _ Subject M SD Perc. M SD Perc. 
4 Mathematics 199.20 13.90 0.49 191.32 15.24 0.28 
5 Mathematics 209.12 15.30 0.50 200.53 16.08 0.29 
6 Mathematics 214.41 15.59 0.49 203.04 15.40 0.23 
7 Mathematics 220.69 17.27 0.51 215.10 17.65 0.38 
8 Mathematics 226.21 18.46 0.50 221.76 18.89 0.43 
4 Reading 196.13 15.98 0.49 191.98 19.06 0.39 
B) Reading 203.81 15.63 0.49 200.82 18.52 0.41 
6 Reading 209.70 15.41 0.49 207.29 18.37 0.43 
7 Reading 213.82 15.58 0.49 211.96 18.36 0.45 
8 Reading 217.64 15.68 0.49 216.40 18.27 0.46 


Note. M=Mean, SD=Standard deviation, and Perc. = Percentile score under NWEA’s 2020 
Norms (Thum & Kuhfeld, 2020). Scores are reported for Grades 4-8 because we are tracking 
cohorts of students who are in Grades 3-7 in 2017-18 into the fall of 2018, so results are only 
reported for the subsequent grade levels (e.g., Grades 4-8). 
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